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It has been shown experimentally that a decimation algorithm based on Survey Propagation (SP) 
equations allows to solve efficiently some combinatorial problems over random graphs. We show that 
■ these equations can be derived as sum-product equations for the computation of marginals in an 

^ ' extended space where the variables are allowed to take an additional value - * - when they are not 

, forced by the combinatorial constraints. An appropriate "local equilibrium condition" cost/energy 

function is introduced and its entropy is shown to coincide with the expected logarithm of the 
number of clusters of solutions as computed by SP. These results may help to clarify the geometrical 
notion of clusters assumed by SP for the random K-SAT or random graph coloring (where it is 
conjectured to be exact) and helps to explain which kind of clustering operation or approximation 
is enforced in general/small sized models in which it is known to be inexact. 
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I. INTRODUCTION 



Recent developments in statistical physics of disordered systems have shown a remarkable convergence of themes 
with other disciplines such as computer science (e.g combinatorial optimization information theory (e.g error 
correcting codes |2j) and discrete mathematics (e.g. random structures 0j|Jil3)- While the study of a typical static 
measure characterizing the slow dynamics of both physical and algorithmic processes is the unifying issue in out-of- 
£j ' equilibrium problems, the study of the geometrical structure of ground states of spin-glass-like energy functions E 
is central to the understanding of the onset of computational complexity in random combinatorial problems. The 
combinatorial problem of satisfying a given set of constraints is viewed in the physics framework as the problem of 
minimizing E and "ground state configurations" , "solutions" or "satisfying assignments" should be understood as 
. synonymous. 

Important in an attempt of providing a complete theory of random combinatorial problems is the notion of pure 
states, or clusters of configurations, on which the probability measure over optimal configurations is assumed to 
concentrate. Recently, a new class of algorithms has been proposed H,ilE| that have shown surprising capabilities 
in dealing with the (exponential) proliferation of clusters of metastable states and therefore in solving random instances 
of combinatorial problems which are difficult to solve for local search heuristics. Such algorithms are based on the so 
called Survey Propagation (SP) equations in which indeed a decomposition of the ground states probability distribution 
- the Gibbs measure - into an exponential number of clusters is assumed from the beginning. The SP equations can 
be viewed as zero temperature cavity equations l20l formulated for single instances at a level equivalent to the one-step 
of replica symmetry breaking (1-RSB) scenario |27j . 

The SP algorithm consists in a message-passing technique which is closely related to another message-passing 
method - known as sum-product or Belief Propa gati on (BP) 0, ^| algorithm - which have shown amaz- 
ing performance for solving the decoding problem |l3l | in error correcting codes based on sparse graph encod- 

mgs Gi m nam mug. 

■ The aim of this study is to discuss the precise (finite size) structure of the SP equations, linking them to the BP 
k^J ' formalism. This is a well defined mathematical issue, independent on the physical origin of the equations. Due to the 
algorithmic relevance of both BP and SP for coding theory and combinatorial optimization, it is a basic question to 
understand what these equations are doing for a finite number of variables N since this is the regime in which they 
are used. 

As we shall see, the SP "algorithmic" equations at finite N are performing a very specific clustering operation over 
the solution space. Moreover, the number of such clusters in the Bethe approximation will be shown to coincide with 
the prediction of the cavity theory. 
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These results will be obtained by showing that the SP equations are the BP equations for a modified combinatorial 
problem. By this mapping we clarify how the hypothesis making BP exact (that is, uncorrelation of distant variables) 
translate onto a condition of uncorrelation of "frozen" variables belonging to different clusters: SP produces a collapse 
of the internal structure of clusters and eliminates correlations among the unfrozen parts. 

We shall present the results in the case of the K-SAT problem even though the method could be applied to any 
discrete combinatorial model defined over locally tree-like graphs. The results concerning the cluster entropy will be 
compared with the prediction of the 1-RSB cavity analysis for random K-SAT. 

The line of reasoning of the paper consists in showing that the SP equations can be re-derived as sum-product 
or BP equations - i.e. simple replica symmetric (RS) cavity equations - over an extended configuration space. The 
definition of this space consists in associating to each binary variable a new extra value "*" which will correspond to 
the possibility that the variable is not forced to take one of the binary values { — l,+l}ina given solution [28|. We will 
introduce a local equilibrium condition (LEC) cost-energy function E derived from E, acting over the extended space, 
together with a (technical) duality transformation needed to preserve the locality of the interactions for implementing 
properly the BP equations. The following two statements will hold: 

(I) Marginals given by the BP equations derived from E coincide with the marginals given by SP on the original 



(II) Bethe approximation to the entropy of E in the enlarged space as computed by BP coincides with the logarithm 
of the number of clusters of solutions - the so called "complexity" - predicted by SP on the original problem. 

The proof of (I) will be achieved by finding a direct connection between quantities ( "messages" ) propagated by 
the two algorithms at each iteration step. We recall that the Bethe approximation to the entropy is exact over trees 
without and with boundary conditions, i.e. with leaf variables taking given values. 

The possibility of interpreting SP as appropriate BP equations may have consequences for their rigorous probabilistic 
analysis, through a proper application/generalization of the known methods for the analysis of convergence of BP 
like equations over random graphs (as it has already been done for problems like the random matching |j|). Some 
preliminary exact numerical results that we give in the concluding section are in support of this possibility. 

Throughout the paper we heavily rely on the notations of refs. for what concerns the SP equations. 



SP and BP (or sum-product) are examples of message-passing procedures. In BP the unknowns which are evaluated 
by iteration are the marginals over the solution space of the variables characterizing the combinatorial problem (e.g. 
binary "spin" variables). According to the physical interpretation, the quantities that are evaluated by SP are the 
probability distributions of local fields over the set of clusters. That is, while BP performs a "white" average over 
solutions, SP takes care of cluster to cluster fluctuations, telling us which is the probability of picking up a cluster 
at random and finding a given variable completely biased (frozen) in a certain direction - that is forced to take the 
same value within the cluster - or unfrozen. 

In both SP or BP one assumes to know the marginals of all variables in the temporary absence of one of them and 
then writes the marginal probability induced on this "cavity" variable in absence of another third variable interacting 
with it (i.e. the so called Bethe lattice approximation for the problem). These relations define a closed set of 
equations for such cavity marginals that can be solved iteratively (this fact is known as message-passing technique). 
The equations become exact if the cavity variables acting as inputs are uncorrelated. They are conjectured to be an 
asymptotically exact approximation over some random locally tree-like structures |9|- 

The if-satisfiability problem (If- SAT) is easily stated: Given N Boolean variables each of which can be assigned 
the value True (1) or False (-1), and M clauses between them, is there a 'SAT- assignment', i.e. an assignment of the 
Boolean variables which satisfies all constraints? A clause takes the form of an 'OR' function of K variables in the 
ensemble (or their negations). A SAT formula in conjunctive normal form over N Boolean variables {<7j = ±1} can 
be written as 



problem. 



II. SURVEY PROPAGATION, BELIEF PROPAGATION AND K-SAT 
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C a = l-E, 
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E a = Y\_S(J a<i ,ai) 
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where 8(x, y) is the Kronecker function (also written as b~ x . y in the rest of the paper) and {C a } are the clauses encoded 
by the parameters J a ^ as follows: J a .i = ±1 if respectively ±<7j appears in clause a (in Boolean notation we would 
have J a .i — —1 (resp. +1) if the Boolean variable Xi (resp. ->afi) appears in clause a). We call E a the "energy" of a 
clause. The symbol i £ a will denote the set of variables participating in clause a. Additionally it will be useful to 
use the symbol aeito denote the set of clauses depending on variable i. The clause size \{i : i £ a}\ will be denoted 
by n a (n a = K for A'-SAT), and the variable connectivity \{a : a £ i}\ will be denoted by m. 

The satisfiability problem consists in determining the existence of an assignment to the Boolean variables which 
satisfies all clauses at the same time, that is such that T = 1. We may write the energy function which counts the 
number of violated clauses as E = E a so that the satisfiability problem becomes finding the zero energy ground 
states of E. The random version of A-SAT corresponds to the case in which the variables appearing in each clause 
are chosen uniformly at random, and negated with probability i. For the sake of simplicity, hereafter we concentrate 
mostly on the 3-SAT case. 

The energy function A of a random 3-SAT formula is a spin glass model defined over a locally tree-like graph that 
can been studied with the techniques of statistical physics of random systems, namely the replica and cavity methods. 

Numerical experiments have shown that a decimation algorithm based on SP equations allows to find satisfying 
assignments of critically constrained random 3-SAT instances - that is random formulas with a = M/N just below 
a critical ratio a c ~ 4.267 where formulas are conjectured to become unsatisfiable with high probability - with a 
computational cost roughly scaling as A log A while the other known algorithms typically take times that are 
exponential in A J2 l| . l22j . According to the cavity - or SP - analysis , in such hard region (more precisely for 
a £ [4.15,4.267] 0|23|) there is a genuine one step RSB phase, in which the space of solution decomposes into an 
exponential number of clusters and where metastable states are even more numerous. 

As discussed in great detail in ref. 0, one crucial feature that comes out from the SP analysis is the distinction 
between frozen and unfrozen variables within the different clusters and we shall introduce a formalism which naturally 
incorporates such phenomenon (see also refs. (24[). 

We want to represent the condition for a variable of being not forced to take any specific value in a given ground 
state (unfrozen) and to this end we consider configuration space of 3— value variables Si £ {— 1,*, 1,} instead of 

(Tie {-1,1}. 

We observe that C a as defined in Eq. can be evaluated also in extended variables: it behaves as if variables with 
the * value could be chosen to the best of —1 or 1 and thus satisfy the clause. This gives the name "joker state" to 
the value *. For a configuration s^ hX ^ such that s\ l ' x ' — x and — Sj for j ^ i call 

Cj*(«) = C B (3) 
and introduce the constrain over {—1, *, 1}" configurations given by 

\. » \\c; c:r • £ s s ^]Jc^ fi-II^ - ') (4) 

The LEC formula derived from T will be defined as 

Q = \{V, (5) 

i 

Note that Vi depends only on (sj)j^ a ,a£i and therefore preserves the "locality" of the structure, if any, of the original 
formula. A solution of the LEC problem is a configuration s = (sj)j e j £ {— 1,*, 1}" such that Q (s) = 1. As a 
particular case, a solution Q(s) = 1 such that Sj £ {±1} is also a solution of T . 

To fix ideas it might be useful to compare the LEC cost-energy function with the original 3-SAT one. To this end 
we adopt the so-called factor graph representation [25| : Given a formula J-, we define its associated factor graph as 
a bipartite undirected graph G — (V; E), having two types of nodes, and edges only between nodes of different type: 
(i) Variable nodes, each one labeled by a variable index in / = {1, . . . , A} and (ii) Function nodes, each one labeled 
by a clause index a £ A (\A\ = M). An edge (a, i) will belong to the graph if and only if a £ i or equivalently 
i £ a. For instance, the factor graph representation of the random 3-SAT problem consists in a bipartite graph with 
A variable nodes having a Poisson random connectivity of mean 3a and M function nodes with energy E a of uniform 
connectivity 3 (a portion is shown in part (a) of Fig^). The extended LEC spin glass energy function reads: 



M N 
a—l i—l 



(6) 
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FIG. 1: (a) Portion of the original factor graphs, (b) LEC graph with 3-state variables and additional constraints Ai (black 
nodes) (c) duality transformation (d) dual graph 



where now E a = 1 — C a is evaluated in the extended configuration space and 

A l = <Ss„* (l - <V\ E i) + 5 ^°° ( E i - E i°) ( ? ) 

with Ef = X)aei(l — Ca' CT ) an d ®( x ) = 1 H x > and otherwise. The factor graph of the LEC has N additional 
function nodes (the Ai terms enforcing the joker condition) that extend over the second neighbors (inset (b) in Fig. 

By inspecting Eq. JSJ) we notice a first problem, namely that we have lost the locally tree-likeness of the original 
graph. There are interactions terms between every (ordered) pair of neighbors variable nodes i,j 6 a (in the original 
graph), and thus for instance every such pair shares two constraints Vi, Vj (making an effective 2-loop). This introduces 
an obvious problem for implementing BP over this combinatorial problem, and moreover would make difficult to 
compare both algorithms, as the underlying geometry is now different. Fortunately, there is an easy (but unfortunately 
notationally somewhat involved) way out. We will group together neighbor variables, effectively performing a sort of 
duality transformation over the graph. We describe the procedure explicitly below (Note that this is a particularly 
simple case of a Kikuchi or "generalized belief propagation" -type approximation |26(). 

We will define: (i.) M multi state variables each one corresponding to a tuple t a = {t\^}nz a (t$ G { — 1, *, 1}) and 
"centered" on a clauses and have (uniform) connectivity n a ((c) in FigQ, and (ii.) N function nodes Xi bp having 
Poisson connectivity, depending on Ti = {t a } ae i and enforcing both the joker state condition as well as identifying 

the values of the single variables shared by different tuples o6i ((d) in Fig^J. An explicit expression of xf^^i) 
(conf. Eq. QJ) is 

xf = £ (ikO (^••llc 'c 1 • E ^ \\ ( r (i-n c H) (8) 

{si} \aei / \ a£i a=±l a£i \ a£i / J 

We shall refer to the BP equations over the dual graph as Dual BP (DBP). 



III. SP EQUATIONS AS BP EQUATIONS OVER THE DUAL GRAPH 

Basic SP and DBP iterations can be thought of as transformations in the space of probability distributions of the 
signs hi = { — 1, 0, 1} of the effective fields acting on the single spin variables and of the tuples t a = { — 1, *, l}™ a in 
the dual graph. In the cavity notation the quantities that are iterated refer to a graph in which a given node and 
all its neighbor nodes are temporarily eliminated (see Fig. ^ (a) and (d)) and all quantities are labeled by oriented 
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indices of the type a — ► i or i — > a where the node on the right of the arrow is the one eliminated. Therefore the 
equations describe a local transformation of some input probability distributions into an output distribution in 
which a characteristic function x eliminates contributions from those combinations of input and output fields or 
variables that violate some kind of local constraints (it is worth noticing that these cavity equations are closely related 
to the iterative local equations of the so called Objective Method |3j of combinatorial probability). Explicitly we have: 

DBP equations: 

p^ita) « e n xf n p s&) (9) 

{U}j£a\i bej\a 



SP equations: [9I Ho| 



p,z a (hj) oc 5>£„(mm) n n (10) 

{h k } b£j\akeb\;j 



where 

x^ a = s hj< * n ( tc;; 1 • E ^ n c r f 1 n °r a ] en) 

6ei\a ff=±l 6ej\o \ 6ei\a / 

Ch clauses are here evaluated in ( (/ifc)fee&Y? > ^i) • 

In order to show the connection between the above equations it is convenient to introduce an auxiliary transfor- 
mation r of a similar type: 

t transformation: 

(*-) «eii (*-> m ( i2 ) 

and 

xj-a= E a^, ff <%v+<w ^,^c^- i c^ i + e ^vcr (1 - cr CT ) (13) 

tr=±l L tr=±l 

C Q terms are evaluated here in t a . 

We will drop now the argument dependence of the measures Pj^ a , P a Si an( l Pj—m an( l ma k e instead explicit the 
dependence on the input probability measures {Pk~^b} , {Pb^j} , {Pj— >o} respectively. 
The connection between DBP and SP can be written as follows: 

P a% ({PU}) = ({P;ia}) (14) 

where both sides of the (functional) equality in turn depend on some arbitrary set of probability distributions {Pk{hk)} 
where k £ b\j for b S j \ a and finally j G a\i. In short, 

pdbp oP r =pr psp (- 15 ) 

In order to check the validity of the above identity we observe that a direct inspection of the composition shows 
that it is true if for every j G a \ i the following condition among the characteristic functions holds: 

e xuxT^ a = e xf p n n *u (i6) 

{h 4 } {t b } bej\akeb\ 3 

In appendix^we display the proof that this identity holds and, as a consequence, that also identity Eq. 1151) is valid. 
Eq. IJ15H in turn implies that 

(pdbpjW o pr = pr Q (p SP )(fe) ; ( 17 ) 
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FIG. 2: The whitening procedure from left to right: the original set of solutions {( — 1, — 1, —1), (1, 1, — 1), (1, 1, 1)} and the set 
of whitened clusters in the final step {( — 1, — 1, —1), (1, 1, *)} 



where the (fc) exponent means composition. This in turn implies that we have a direct step-by-step connection 
between the elementary quantities used in the DBP equations and those used in the SP equations: convergence is 
obtained simultaneously and Eq. (|15|) holds for the respective fixed points. It is straightforward to compute from 
the DBP equations the marginals P i p (sj) of the single variables as a marg inalization of P£ b P (t a ) for some a £ i 
with respect to all other variables in the clause, (on a fixed point, it doesn't matter which a £ i one chooses). One 
finds that the marginals predicted by DBP are in one to one correspondence with the local fields given by SP, that is 
Pf bp {si = —1, *, 1) coincides respectively with P- p {Hi = —1, 0, 1) (see refs. [9llTo|V 

A. Clustering and whitening 

The marginals over {1, *, — 1}^ given by SP/DBP acquire a computational/physical significance once we interpret 
what solutions of combinatorial problem defined by Eq. (JSJ mean in term of clusters (or groups) of solutions of 
the original problem defined by Eq. QJ. We will first define the Hamming distance between configurations s,t £ 
{1, *, —1}", H(s,t) = \{i : si ^ ti}\ and an ordering relation over { — 1, *, 1} configurations: if s,t £ {1, *, —1}™ we say 
that s < t iff ti 7^ Si implies that ti = *. For instance, (0, 1) < (0, *) and (1, 1, 1) < (1, *, *) but (0, 1) ^ (1, *). 

We will say that a configuration s £ {±1}™ is contained in t £ if s < t. In this sense, "clustering" would mean, 
starting with some set S C {±1}™ of solutions of the original combinatorial problem, to find some set T C {1, *, — 1}™ 
such that every s £ S is contained in some t £ T. Of course, one would like to do so in some maximal way, but 
satisfying some kind of separation between different clusters. 

One trivial observation about the set Q — 1 is that solutions are by force separated, in the sense that H(s, t) > 1 
if Q{s) — Q(t) = 1 and s ^ t. To prove this, suppose that H(s,t) = 1. If their difference comes because Si = ±1 and 
ti = * then by force one of Vi(t) or Vj(s) is clearly violated. If on the contrary, it comes because Si = 1 and ti = — 1 
or viceversa, then by force both of Vi(t) and Vi(s) are violated and the only possible "correct" value for is *. 

A more important observation is that every solution of T = 1 is contained in a solution of Q = 1 with the minimal 
number of *, and that solution can be easily found. Take a solution x of J- = 1, and suppose that Q = 0, Choose a Vi 
such that Vi — 0. It can be easily seen that by replacing Xi by *, then Vi becomes 1. Then we pick another violated 
constrain and repeat the process, until Q = 1. We will call the resulting configuration w{x) (this procedure has been 
already used under the name of whitening in the context of graph coloring by G. Parisi in [24|]). It is easy to prove 
that the result of this procedure does not depend on the order in which you pick variables violating nodes Vi (the 
proof being that any violated Vi will continue to be violated in the procedure, exactly until we switch x\ to *), and so 
w{x) is uniquely defined. Note that two configurations x, y at Hamming distance H (x, y) = 1 will have w(x) = w{y) 
and so every solution in a fixed connected component of the solution space will end up inside the same "cluster" . An 
example of the whitening procedure for some set of solutions is depicted in Figure @ ■ An interesting point of view 
is that if one tries to build from scratch a Hamiltonian to describe the behaviour of the outcomes of the whitening 
procedure of some SAT formula, Eq. (JSJ comes naturally. 

The reader should note however that the presented definition of clustering is far from perfect in the worst case: there 
is a number of systematic errors produced by the whitening. For instance, in Figure © we can see one cluster claiming 
an uncorrectly large volume. And there is of course also another problem: unfortunately, there is no warranty that 
the sole solutions of Q = 1 are the ones of the whitening, and in fact small counter-examples can be easily constructed. 
Numerical work is being done to ascertain a quantification of these two types of errors ( 32] ) . 

IV. ENTROPY AND COMPLEXITY 

The equivalence between the DBP marginals and the SP local field probability distributions has the direct con- 
sequence that the Bethe approximation to the entropy on the dual graph, S dbp , coincides with the logarithm of the 



Jf 



FIG. 3: A systematic error of the whitening w((l, 1, —1)) (the dark solution in the left). From left to right: the original sets 
of solutions {(1, 1, —1), (1, 1, 1), (1, — 1, 1), ( — 1, — 1, —1)} and first step (1, 1, —1), second step (1, 1, *), third step {(1, *, *)} and 
final step {(*, *, *)} 



number of clusters of solutions predicted by SP, the so called complexity S. 

On general grounds the Bethe approximation to the entropy of a problem is exact if correlations among cavity vari- 
ables can be neglected (i.e. the global joint probability distribution takes a factorized form). This is certainly true over 
tree graphs and it is conjectured to be true in some cases for locally tree-like random graphs in the limit of large size (one 
informal explanation is that distance between cavity variables diverges with probability tending to one) . Factorization 
of marginal probabilities over our dual factor graph amounts at writing P({t a }) — Y\ ieI Pf bp (Ti) Y\ aeJ i[Pa bp (ta)] 1 ^ na 
where Pf bp (Ti) is the joint probability distribution of the triples connected to node i (Tj = {h}bei) and P^ bp (t a ) is 
the single triple marginal. Under this condition the entropy reads 

S=-EE P^^) J °g Pf^Ti) P' bP (ta) log Pt bP {ta) ■ (18) 

* {T,} " {ta} 

Showing S = £ is a straightforward calculation that we report in the appendix. It requires to express the entropy 
in terms of the cavity fields given by SP exploiting both Eq. H15fl and the fixed point conditions. One finds 

s = J2 log a - E ( n « - !) lo § c - - E E lo s D ^ ( 19 ) 

i a i a£i 

where the three normalization constants are defined by 

c * = EII p ^(^)»( r *) ( 2 °) 

{Ti} aei 

C - = E E IT *J-« (Ma) (21) 

*a {hj}jea 

D «^i = EE II p j^(hj)xUa(hj,t a ) (22) 

to {h s }jea\i 

These constants are not independent and the explicit expressions of the first two are sufficient for writing S in terms 
of SP quantities: 

°« = e n p ^ fo) e n & > *«) ( 2a ) 

{hj}jea {t a }jEa 

= 1 e n p ^ (m ( 1 - e n o ) (24) 

= 1 II'' -(^,i) (25) 
TT" 

= I-TT7 V (26) 

11 (it +n° +n u ) 

where we have borrowed the notation of Eq. (18) in |icj . For computing a we first notice that 

P^i (t a ) = D a ^ E xUa (*-' *i) II P ^ (2 7 ) 
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so that Eq. lj2"fj|) reads 



aei {HjXXj} a jea\i 

= u d <~* e *?wn n p^ihj) 

a£i {Hi} a j£a\i 

= n^(n+ + n° + n-) (28) 



a£i 



in the notations of Eq. (21) in [To| . Finally, plugging these expressions into Eq. (|19fl and calling 

Wi = n+ + n, + nr 



t ■ — TT S 4- TT° + TT M 



y^ a = n^ a (29) 

we get from Eq. I|19|) 



5 = ^logu; 4 -(n Q -l)^log l-n^L (30) 



j£a 



In this expression, Wi represents the probability the local field acting on the spin variable i does not produce a 
contradiction and 1 — v x i ^ a is the probability that the cavity fields satisfy clause a. 
We recall that the expression of the SP complexity E defined in Eq. (25-27) in is 



E ( X ~ n< ) lo S w * + E log ( II x ' l ^ a ~ II Vl ^ a ) 

i a \i£a i£a J 

e io s m; ~ e e i ° gm + e iog ( n Xi ^ a ~ n Vi ^ a ) 

i a i£a a \i<Ea i£a / 



(31) 



Despite their different look, it turns out that Eq. (|30|l and Eq. I|31|) are identical if evaluated in a fixed point of the 
SP equations. Their difference 



E-S = ^ J-^Tlog^+^logf ) -E 1 ^*— 



(32) 



is zero since in the fixed point every term inside the curly brackets vanishes: using Eq. (17) in flOj | we have that 

Va^i = Hjea\i T~ ' Le - Iljea = Wa-^i^tzt for eVer y * G a &nd henCe 



u og (i-n— i =e i °s( 1 -^— ) 



(33) 



A simple calculation shows that Wi = x a ^i — r\ a ^iVa^i for every a £ j and therefore we get E = S 1 as desired. 



V. DISCUSSION AND CONCLUSIONS 



In this work we have shown by elementary means that the SP equations can be interpreted and derived as sum- 
product equations for the marginals over a modified combinatorial problem. An important consequence of this fact 
is a clarification of the hypothesis behind the algorithm. It is to be expected that the essential hypothesis making 
sum-product to work is the uncorrelation of the marginals of distant (or cavity) variables. Under the shown mapping, 
this directly implies that the hypothesis behind SP (and in a way, of its definition of clusters) is the uncorrelation of 
the frozen part of distant variables, that is the uncorrelation between different clusters. 

Under this light one can think of the SP procedure of obtaining E from £ as a way of collapsing the internal 
structure of pure states: the resulting problem Q has many pure states but with zero internal entropy. Note that this 
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is a completely different limit case with respect to the "one pure state" in which BP (more precisely DBP) is shown 
to work correctly and to predict an accurate entropy (which we remind is the complexity of the original E). 

As far as the connection between solutions of the modified problem and the original one is concerned, things are 
particularly simple over tree factor graphs (see also ^} f° r results concerning propagation of messages): Indeed, for 
any fixed boundary condition (i.e. an assignment for the leaf variables), there is at most one solution with E = 0, 
and it is easy to prove (see appendix that all solutions of E — correspond to the same connected component of 
the solution space (i.e. every two solutions can be joined by a path of solutions in which successive configurations in 
the path differ by exactly one spin flip). 

The situation on loopy graphs (corresponding for instance to random formulae) is obviously more complicated. 
A coherent interpretation would be that not only the recursive DBP / SP equations themselves are accurate in a 
probabilistic sense (i.e. when the factorization of the corresponding input joint probability is sound) to compute the 
statistics of the ground states of E, but also that the exactness of the interpretation of the ground states of E in 
terms of clustering of the ground states of E relies on this hypothesis being true. 

To this extent we mention that exact enumerations on a large number (thousands) of small random 3-sat formulas 
(up to N = 100) showed that all the zero energy configurations of E which are stable under SP iterations can be 
extended to real solution of the original problem. Spurious ground states (i.e. configurations that are not extensible 
to real solutions) do exist with a non negligible probability for small N, however they turn out to be always unstable 
fixed points of SP , that is unsat configurations which are irrelevant for the SP marginals [12 ■ While such a result 
was expected to hold for tree- like graphs, it is somewhat surprising to observe it numerically on small, loopy, random 
factor graphs. The robustness of such result calls for a finite N probabilistic analysis which would represent a building 
brick for the rigorous analysis of SP (of course, small ad-hoc counterexamples on improbable formulae can be easily 
constructed). 

As a concluding remark we notice that the discussed formalism can be generalized to take care of the non-zero 
energy regime where not all constraints can be satisfied simultaneously ( "frustrated" case) . The LEC energy function 
takes the form E = A J2aeA Ea+J2iei -^*' wnere ^ plays the role of the so called Parisi re- weighting parameter |20j . 
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APPENDIX A: PROOF OF EQUIVALENCE 



For the LHS of Eq. {TSJ we have: 



lihj = a e {±1} then 




(Al) 



If hj = * then 





(A2) 



<T=±1 



bej\a 



Summing up both products and regrouping the LHS of Eq. <|16f) reads: 




(A3) 



where Cb for b E j \ a is evaluated here in ( {hk}keb\j, ta ) and C a is evaluated in t a . 
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For the RHS of Eq. I|16(l we first notice that as the Xj bp term includes IlaGj ^t u) s we w * u simply replace all 
occurrences of and Sj variables by and drop the outer sum and the product term itself. For instance, the sum 
over {tb}tej thus reduces to a sum over 

{{t^He&Yp*^}- Let ' s evaluate the RHS of Eq. fifty on the three possible 

values of r a ': 

If ti j) = * then by Eq. (SJ xf P = Ubej ^b'^Cfe' 1 - Moreover, just by looking at its definition Eq. JT3J), one finds that 
in Xk^b an C terms are equal to 1 since their j coordinate = to ' is *. Then Xfc-»b = <V fc ) h an d the RHS of 
Eq. Ijl6|) becomes 

cr 1 ^' 1 II or 'or II hfc (A4) 

b£j\a k£b\j 

which is exactly the term in Eq. (|A3|I corresponding to = * (remember that Cb clauses here are evaluated in %)■ 
If ta — cr G {±1} then it is convenient to break xf P m two addenda: 

bej bej 

so that the RHS of Eq. (|16J) becomes 

c « n (e^ n x3u 6 ) -^^r ff n fe^ ff n m 

b€]\a \{t b } keb\j J bej\a \{t b } keb\j J 

Finally, both sums can be computed explicitly and the result is again exactly the corresponding term in Eq. JSHJ. 
This ends the proof of the identity Eq. (|15|) . 



APPENDIX B: COMPUTATION OF THE ENTROPY 

For simplicity of notation, in what follows we write P a (t a ), P a ^i(t a ), Pi(Ti) and Xi(Ti) in place of 
^(*»), i7*(2i) and xf p (T0 respectively and P^ a (h z ) in place of P£ a (hi). 

To compute the entropy l|18|) we first need 

Pa(ta) = C-^W^aih^WxUaitaM) 

{hi} i£a i£a 

i€a {hi} 



Thus calling 



we have that 



fa^i = Pi^a (hi) xUa (*«. h i) ( B1 ) 
{hi} 



J2 P a(ta) log Pa{t a ) = ~C~ 1 log C a + ^ P a (t a ) ^ log f a _^ 

{t a } {t a } *£a 

= -c- 1 logc a + E E P <^) lo S^ ( B2 ) 

i£a {ta} 



Writing w^, = E{t a } p a{ta) log/a^i we get 



E^-^E"^ = EE E 
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= EE E T, p M l °sf a ^ 

i aei jea\i {t a } 

= EEE P «(*«) II l °sf^ 

i aei {t a } jEa\i 

= EEE E pmU^f^ 

% aei {t a } {t b } beAa jea\i 

= EEE p ( T *) lo s II (B3) 

i aei {Ti} j£a\i 

The term inside the logarithm above reads 

n f«r-i = e n x-^a (*a,fci) n = ^- p ^ (*«•) ^ 

j£a\i {hj}jea\i j£a\i 

where D a ->i is an appropriate normalization constant. Going back to Eq. I)B3J| . we have 

EK - J ) E = - E E ^6 A>-< + E E E p ( T <) ^gPa^(ta) (B5) 

The second term in the right-hand side equals 

53 E p ( p *) iog]I p ^(^) = E E p * ( p ') lo sxi(7i) n *W*a) 

i {Tt} a£i i {Ti} aei 

= E E p i ( T *) lo § o< ( T *) 

= EE p '( T i) lo s p <( r i)+EE p i( T ') lo w (^6) 

* {Ti} * {Ti } 

where in the second step above Xi(Pi) has been artificially multiplied inside the logarithm (we can do it because there 
is a P t {T l ) outside) and P^T,) = j-Q l (T,). Eqs. l|B5 )l .l|BB )l give: 

53k - i) 53 ^ = - 53 53 logiv^ + 53 53 p t m log^cro + 53 i ogCi (B7) 

Going back to the first expression of the entropy Eq. (|18fl . and using Eq. (|B2|) and Eq. ijEj we get: 

s = - 53 53 Pi{Ti) bg Pi (^+53(^-1) 53 p a {t a ) bg p a (t„) 

1 {T,} a {ta} 

= 53 l 0gCl - 53 53 P(Ti) ]oeQi(Ti) + 53 (n„ - 1) 53 P * (*») iog-Pa^a) 
i {T s } a {ta} 

= 53logc i -53(n a -l)logc a -5353log J D a ^ (B8) 
where the constants are defined in Eqs. (|20I22|) . 

APPENDIX C: TREE FACTOR GRAPHS 

The argument turns out to be similar to the one given in an analogous "tutorial" appendix in ref. |3l| for the 
Vertex Cover problem. 

We will first build a reference solution x, and then show that every solution of E — is connected to it. x will be 
built from the leaves to the root. Suppose the variables are labeled in an ordering that respects distances to the root, 
such that the first ones are the leaves and the last one is the root. In such an ordering, the parents (resp. child) of i 
are neighbors with labels j < % (resp. j > i). We will fix xi iteratively: once Xj for j < i are fixed, all parents of j are 
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fixed; then for Xj there are two possibilities: either its parents force it to take a specific value, or they don't. In the 
first case we chose to take the forced value; in the second one we chose the value that satisfy the child clause. Now 
we can show that x is connected with every other solution s (and thus every two solution are connected). It is easy 
to see that the configurations y' fe ' defined by = sj if j < k and yj. = Xj if j > k form a path of configurations 
connecting x and s. Clearly yW = x and y' n ) = s. Also they are all solutions, since if y( fe ) is a solution, then clearly 
y( fe +!) i s a i so a solution: if they are different it is because yfc 1 ] 1 has been chosen to satisfy the child clause (and it 
was not forced from parents in s and thus neither in y( k + 1 '). 

We can now look for solutions of E on a satisfiable tree (with boundary conditions). Let's start with a free-boundary 
tree with 2 and 3-clauses: it is easy to see that the solution with all * assignments has E — 0. It is also clearly unique: 
suppose that there is a solution with some variable set to a =/= *. Then there is forcefully one of its neighboring 
clauses in which the two (or one) remaining variables are fixed in order to not satisfy the clause. Repeating again the 
argument recursively for one of them, we can get a never-ending path of fixed variables in the tree. But as a trees 
have no loops, this is a contradiction. 

There is also exactly one such solutions for a satisfiable tree with boundary conditions (if we disregard Vi constraints 
on the variables with assigned boundary values). We will build it explicitly using the so-called unit clause propagation 
(UCP). The UCP procedure consists in removing (in this case starting from the boundary) every fixed variable by 
(a) removing all clauses satisfied by the variable and (b) removing the variable from all clauses in which it appears 
without satisfying the clause, (if the original tree is satisfiable, no 0-clause can appear in this erasure step). Then 
every possibly appearing 1-clause is taken and its variable fixed in order to satisfy the clause, and the procedure 
starts again from the beginning until no more 1-clauses show up. The resulting graph is boundary-free and with no 
Tclauses. 

The promised solution will be built by taking all variables fixed by UCP with their assigned value, and by assigning 
the value * to the remaining ones. The resulting configuration x has E(x) — 0. Clearly the constraints Vi (see 
Eq. J3J) are satisfied by x for all i fixed by UCP (because they are "frozen" by their neighbors). We easily see that 
this partial assignement is the unique one that can give E = 0. Using the fact that the subgraph produced by UCP 
has no boundary condition and that the unique solution for E = on that subgraph is the all-* one, we see that the 
proposed configuration is indeed the unique solution. 

Note also that every solution of E = will coincide with i in the —1, 1-assigned variables of the latter, because these 
variables were fixed by UCP and thus are forced in every satisfying configuration. Moreover, if one takes an index i 
such that ii is *, then there is at least one solution of E(s) — with Si = 1 (resp. —1): by fixing Sj and applying 
again UCP one cannot get any contradiction (i.e. a 0-clause) because the subgraph has no loops nor 1-clauses. The 
remaining graph is still loop-free, and thus trivially satisfiable. 
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